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Abstract 

Objective. The aim of this study was to develop and validate an index to assess the implementation of quaUt^r management 
systems (QMSs) in European countries. 

Design. Questionnaire development was facilitated through expert opinion, literature review and earlier empirical research. 
A cross-sectional online survey utilizing the questionnaire was undertaken between May 2011 and February 2012. We used psy- 
chometric methods to explore the factor structure, reliability and validitjr of the instrument. 

Setting and participants. As part of the Deepening our Understanding of Quality improvement in Europe (DUQuE) project, 
we invited a random sample of 188 hospitals in 7 countries. The quality managers of these hospitals were the main respondents. 

Main Outcome Measure. The extent of implementation of QMSs. 

Results. Factor analysis jdelded nine scales, which were combined to buUd the Qualitjr Management Systems Index. Cronbach's reli- 
ability coefficients were satisfactory (ranging from 0.72 to 0.82) for eight scales and low for one scale (0.48). Corrected item-total 
correlations provided adequate evidence of factor homogeneitjr. Inter-scale correlations showed that every factor was related, but 
also distinct, and added to the index. Construct validity testing showed that the index was related to recent measures of qualitjr. 
Participating hospitals attained a mean value of 19.7 (standard deviation of 4.7) on the index that theoretically ranged from 0 to 27. 

Conclusion. Assessing QMSs across Europe has the potential to help policy-makers and other stakeholders to compare hospitals 
and focus on the most important areas for improvement. 

Keyv^ords: quaMt}' management, hospital care, surveys, patient safet)', health care system 



Introduction 

In a recent review on instruments assessing the implementa- 
tion of quality management systems (QMSs) in hospitals, the 
authors conclude that hospital managers and purchasers 
would benefit from a measure to assess the implementation of 
QMS in Europe. The results of the review show that there is 
currendy no weU-estabHshed measure that has also be used to 
assess the link between qualitjr management at hospital level, 
quaMt)' management activities at departmental level and patient 
outcomes [1]. In the context of the European cross-border 



directive and the Council Recommendation on patient safety, it 
is even more important to measure and compare the imple- 
mentation of QMS across countries to get insight into existing 
prerequisites for safe patients' care and possible gaps in the 
qualitjr management within or between countries. QMSs defin- 
ition used in this article is 'as a set of interacting activities, 
methods and procedures used to monitor, control and 
improve the quaMtjr of care' [2] . 

In the recent review, 18 studies to assess the implementation 
of QMSs have been described. Only nine of these studies 
reported methodological criteria in sufficient detail and were 



^The complete details of The DUQuE Project Consortium are given in Appendix. 
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rated as good [1]. Only two of them have been used in several 
European countries, e.g. the European Research Network on 
Quality Management in Health Care (ENQuaL) questionnaire 
for the evaluation of quaUt}' management in hospitals [3, 4], and 
the Methods of Assessing Response to Quality Improvement 
Strategies (MARQuIS) questionnaire and classification model 
for quality improvement systems [5]. Despite their good evalu- 
ation, both instruments have important limitations. The 
ENQuaL questionnaire was developed in 1995 and does not 
cover more recent quality' management topics, such as 'use of 
indicator data' and 'learning from adverse events' [2]. The 
MARQuIS questionnaire is very long (113 items) and focusses 
mainly on leadership (36 items), policy, planning, documents 
(20 items), quaUtjr strategies (laboratory) (20 items) and structure 
(19 items), but less on the evaluation of care processes by indi- 
cator data. The latter is an important step in the quaUt}' improve- 
ment cycle [5]. 

The objective of this study was to develop and validate an 
up-to-date and more concise survey instrument to assess 
hospital QMSs in European countries, and to compute an 
index — the QuaUt}' Management Systems Index (QMS!) — 
representing its developmental stage. Specifically, we report on 
its structure, reliability, validity and descriptive statistics of the 
QMSI and its scales. 



Methods 

Conceptual considerations 

A broad range of activities can be used by an organization to 
maintain and improve the quaHtjr of care they deliver. These 



activities might change over time because of new evidence, 
changing expectations of the public or new (national) regula- 
tions regarding accountability. When developing a multi-item 
measurement instrument, we need to know the underlying re- 
lationship between the items (quaKt)' activities) and the con- 
struct to be measured, e.g. the QMS. The new instrument has, 
like the earlier developed ENQuaL and MARQuIS question- 
naire, partiy been based on the nine enabler and resulting 
themes of the existing theoretical framework of the European 
Foundation of Quality Management model (EFQM) [6], but 
some of the questionnaire items had to be changed to repre- 
sent actual developments in quality management practice. 

Development of the instrument 

The questionnaire was a web-based multi-item and multi- 
dimensional instrument to assess the development of QMSs 
in hospitals. The aim of the questionnaire was to focus on the 
managerial aspects of quality management such as policy 
documents, formal protocols, analyzing performance and 
evaluating results, and not on leadership, professional and 
patient involvement or organizational culture, as these are 
different theoretical concepts within the Deepening our 
Understanding of QuaUtjr improvement in Europe (DUQuE) 
framework (Fig. 1) and which are assumed to influence the im- 
plementation of QMS. The literature review revealed that 
earlier studies have distinguished six domains of quaUtjr man- 
agement, e.g. procedures and process management, human re- 
source management, leadership commitment, analysis and 
monitoring, structures and responsibilities and patient involve- 
ment. Most of the instruments do not cover all the domains. 
If they do, they have a large number (179) of items [1]. 
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Several steps were applied to develop a more concise instru- 
ment that stiU covers the most important domains of QMS 
presented in the literature. 

To select items from earlier questionnaires (ENQuaL and 
MARQuIS) [2, 5] or develop new items for the DUQuE ques- 
tionnaire, we first used the expert opinion of other DUQuE 
project members. They considered the most relevant and pos- 
sibly most influencing activities for the improvement of 
patient-related outcomes («=10). The experts have a long 
history in healthcare, especially in quality management in the 
various countries. For a concise instrument, only the most 
relevant activities are important. Second, items related to the 
more managerial focal areas of the theoretical framework of 
the EFQM model were selected (policy documents, human 
resources, processes and feedback of results such as patient 
and professional experience, comparison of clinical and soci- 
etal performance). The wording of the items and the answer 
categories were compared with accreditation manuals and the 
review on existing instruments. In the end, most questionnaire 
items for the new instrument came from the ENQuaL and 
MARQuIS questionnaire, but because of our focus on man- 
agerial aspects of quality management, not all focal areas of 
these instruments were selected. Finally, the answer categories 
of all items were standardized with a focus on the extent of im- 
plementation and four answer categories. 

The questionnaire was first developed in English and was 
translated into seven languages using a forward— backward 
translation process for validation. Respondents could rate each 
item on a four-point-Iikert-type scale, with answer categories 
ranging from 'Not available' to 'FuUy implemented' and from 
'Disagree' to 'Agree'. 

The content validity of the final questionnaire used in the 
DUQuE project was approved and judged completely by the 
10 experts from different quality research areas involved in 
the project who were not involved in the QuaUtjr Management 
Systems Index (QMSI) development. The questionnaire con- 
sisted of 56 items divided over 5 dimensions: quality policy 
(10 items), quaUtjr resources (9 items), performance manage- 
ment (7 items), evidence-based medicine (13 items) and internal 
quaUt)' methods (17 items). 

Setting and participants 

The study took place within the context of the DUQuE 
project that ran from 2009 to 2013 [7, 8] in 7 European coun- 
tries: France, Poland, Turkey, Portugal, Spain, Germany and 
Czech Republic. These countries represent the diversity' of 
Europe (e.g. countries from the East/West, North/South, re- 
gional/national healthcare system, system in transition/longer 
established system). In each country, 30 hospitals were ran- 
domly recruited if they had >130 beds and were treating 
patients with acute myocardial infarction, hip fracture and 
stroke and handled child deliveries. The conditions were 
chosen for their high financial volume, high prevalence of the 
condition and percentage of measureable complications, and 
the different types of patients and specialists they were cover- 
ing [7, 8]. Of the 210 approached hospitals, 188 were able to 
participate (89.5% response rate). The DUQuE QMSI 



questionnaire was administered online to the qualitjr managers 
of the 188 participating hospitals (response N=183; 97%). 
A quality manager of a hospital was defined as the person who 
is responsible for the coordination of quality improvement ac- 
tivities. He/she should have a good overview of all activities 
toward quality improvement (questionnaire instruction). The 
quality manager was allowed to ask other people in the hospital 
if he/ she was not sure about the right answer, but only one 
questionnaire per hospital was expected to be filled in. The in- 
struction also said that it was not necessary for a hospital to 
have all activities mentioned in the questionnaire and that it 
was expected that hospitals would be in different phases of im- 
plementation for different activities. 

Ethical approval was obtained by the project coordinator at 
the Bioethics Committee of the Health Department of the 
Government of Catalonia (Spain). 

Data colkction. Respondents who participated in the DUQuE 
project were invited by a letter and personally by the country 
coordinator. Questionnaires were completed anonymously and 
direcdy entered in the online data platform. The data were 
collected between May 2011 and February 2012. AH participants 
were sent passwords to access the web-based questionnaire and 
sent reminders. 

Statistical analyses. We began by describing the hospitals and 
quality managers that provided responses to the main 
questionnaire used to develop the index. Next, we used 
psychometric methods to investigate the structure, reliabilitjr and 
validity of the QMSI instrument. We assumed that our ordinal 
data approximated interval data and conducted exploratory 
factor analjrsis and confirmatory factor anal}fsis, reliability 
coefficient, item-total scale correlation and inter-scale correlation 
analyses [9—11]. These were done separately for each of the 
theoretical themes. We explored the factor structure of the 
questionnaire using split-file principal component analysis with 
oblique rotation and an extraction criterion of eigenvalues of >1 
while requiring three or more item loadings. Items were grouped 
under the factor or scale where they displayed the highest factor 
loading. Only items that had loadings of at least 0.3 were assigned 
to a factor [10]. Confirmatory factor analysis was then used on 
the second half of the sample to determine whether the data 
supported the final factor structure. A root mean square residual 
of <0.05 and a non-normed fit index of >0.9 indicated good fit 
of the scale structure to the data. We then performed reliability 
analysis using Cronbach's alpha where a value of 0.70 or greater 
indicated acceptable internal consistency reliabilitjr of each scale 
[12, 13]. We also examined the homogeneity of each scale using 
item-total correlations corrected for item overlap. Item-total 
correlation coefficients of 0.4 or greater provided adequate 
evidence of scale homogeneitjr. Finally, we assessed the degree of 
redundancy between scales by estimating inter-scale correlations 
using Pearson's correlation coefficients, where a correlation 
coefficient of <0.7 indicated non-redundancy [11, 14]. 

Once we had a final factor structure, we computed the 
score for each of the scales by taking the mean of items used 
to build the scale. We used appropriate multiple-imputation 
techniques to handle missing data for hospitals with missing 
data for four or fewer scales used to buUd the final QMSI [15]. 
The scores of the extracted scales of our analysis were then 
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summed in order to construct the final QMSI. We subtracted 
tlie number of factors or scales from this sum in order to 
bring the lower bound of the scale down to zero. 

In order to validate our instrument, we further examined 
correlations with two other measures of qualitjr management 
based on on-site visits by external auditors. These other con- 
structs were the Quality Management Compliance Index 
(QMCI) and the Clinical Quality Improvement Index (CQII) 
[16]. The QMCI measures the compliance of healthcare pro- 
fessionals, managers or others responsible in the hospital with 
quality management strategies. The CQII measures the imple- 
mentation of clinical quality strategies by healthcare profes- 
sionals. Both measures are based on on-site visits of external 
auditors and are described separately in this supplement [16]. 

We used Pearson's correlation coefficients to assess the rela- 
tionship between QMSI, QMCI and CQII, deeming coeffi- 
cients between 0.20 and 0.80 as acceptable [10, 14, 17]. If the 
QMSI measures the implementation of QMS in hospitals, it is 
expected that there would be a positive non-coUinear relation- 
ship between the QMSI and the two more independent mea- 
sures of quality management: QMCI and CQII. Because only 
some parts of the content of the three instruments overlap, 
the coefficients wiU not be very high. 

AU statistical analyses were carried out in SAS (version 9.3, 
SAS Institute, Inc., NC, USA. 2012). 



Results 

Participants 

A total of 188 hospitals participated in the DUQuE project. 
Quality managers of all the hospitals responded to the ques- 
tionnaire, but five qualit}' managers provided not enough data 
to calculate the nine scales and the QMSI. Background 
characteristics of the participating hospitals and the quality 
managers who filled in the questionnaire are given in Table 1 . 



Structure, reliability and validity 

Table 2 gives an overview of factor loadings, Cronbach's 
alphas and corrected item-total correlations for each of the 
nine scales retained from factor analysis that were used to 
buUd the QMS index. These nine scales were quality policy 
documents (three items), quality monitoring by the board (five 
items), training of professionals (nine items), formal protocols 
for infection control (five items), formal protocols for medica- 
tion and patient handling (four items), analyzing performance 
of care processes (eight items), analyzing performance of pro- 
fessionals (three items), analyzing feedback and patient experi- 
ences (three items) and evaluating results (six items). We 
eliminated 10 of the original 56 items in the questionnaire due 
to low factor loadings. As seen in Table 2, factor loadings 
ranged from 0.34 ('benchmarking') to 0.89 ('professional train- 
ing in quality improvement methods'), with most items achiev- 
ing acceptable factor loadings (> 0.40). Confirmatory analysis 
supported this final structure (not reported here). 
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Table I Characteristics of hospitals and quaHtjr managers 
(N= 183) 



Hospital characteristics 



Teaching status, N (%) 

Non- teaching 106 (57.9) 

Teaching 77 (42.0) 

Ownership, N (%) 

Private 32 (17.4) 

PubHc 151 (82.5) 

Number ofbeds,N(%) 

<200 18 (9.8) 

200-500 78 (42.6) 

500-1000 60 (32.7) 

>1000 27 (14.7) 

QuaHt}' manager characteristics 
Sex, JV (%) 

Male 60 (32.7) 

Female 123 (67.2) 

Age (years), mean (SD) 44.6 (8.6) 

Age missing, N (%) 3 (0.0) 

Number of years affiliated with the 13.2 (9.6) 
hospital, mean (SD) 

Hospital years missing, N (%) 7 (0.0) 
Number of years as quality manager, mean (SD) 4.6 (3.2) 

Job years missing, N (%) 5 (0.0) 



Cronbach's alphas for internal consistency reliability were 
satisfactory for all scales (Cronbach's alpha = 0.72—0.87) 
except 'analyzing feedback & patient experiences' (Cronbach's 
alpha = 0.48). Based on the theoretical importance of feed- 
back of patient experiences and benchmarking, we decided to 
keep this scale in the QMSI. The item-total scale correlations 
were acceptable within the range of 0.20 to 0.80. The correl- 
ation coefficients for items in the scale 'feedback of patient 
experiences and benchmarking' were consistentiy lower than 
those for the other scales. As shown in Table 3, the inter-scale 
correlation ranged from 0.1 1 (between 'feedback of patient ex- 
perience' and 'formal protocols for infection controls') to 0.70 
(between 'evaluating results' and 'analyzing performance of 
care processes'). For all scales, each inter-scale correlation was 
below the pre-specified 0.70 threshold and deemed acceptable. 

The vaUdit)' of the QMSI was further explored by analyzing 
its correlations with two other measures of qualitjr manage- 
ment, namely the QMCI and the CQII. Correlation coefficients 
were within the acceptable range of 0.20 to 0.80 (Table 4). 



Descriptive statistics for the QMSI and its scales 

Descriptive statistics of items used to build the scales and the 
index are provided in Table 5. AU items of the questionnaire 
were on a Likert-type response scale from 1 to 4. The average 
score on the individual items was ~3, with a lower average 
score for items related to the analysis of the performance of 
professionals. Some floor and ceiling effects were found. 
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Table 2 Factor loadings, Cronbach's alpha and corrected item-total correlations (N — 181) 



Scale and items 



Factor 
loadings 
on primary 
scale 



Internal 

consistency 

reliability: 

Cronbach's 

alpha 



Corrected 
item-total 
correlation 



Quality polig documents 0.75 

1 . Written description of a formally agreed quality policy 0.81 7 0.655 

2. Quality improvement plan at hospital level (translation of the qualit}' objectives 0.847 0.718 
into concrete activities and measures designed to realize the qualit}' poUqr) 

3. Balanced score card (an overview of key quality measures focusing on clinical 0.424 0.381 
outcomes, finances, human resources and patient satisfaction) 

Quality monitoring by the board 0.87 

1 . The hospital (management) board makes it clear what is expected from care 0.780 0.730 
professionals in regards to quality improvement 

2. The hospital (management) board has established formal roles for quaUtjr 0.559 0.527 
leadership (visible in organizational chart) 

3. The hospital (management) board assesses on an annual or bi-annual basis 0.791 0.739 
whether care professionals comply with day-to-day patient safety procedures 

4. The hospital (management) board knows and uses performance data for 0.776 0.715 
quality improvement 

5. The hospital (management) board monitors the execution of quality' 0.870 0.809 
improvement plans 

Training of professionals 0.84 

1. Care professionals are trained by the organization to do their job 0.539 0.518 

2. Care professionals are trained in teamwork 0.578 0.531 

3. Middle management is trained in quality improvement methods 0.825 0.702 

4. Care professionals are trained in quaUt}' improvement methods 0.889 0.781 

5. Care professionals are trained in patient safety procedures 0.751 0.678 

6. Care professionals follow at least one training session a year to further 0.509 0.479 
develop their professional expertise 

7. Care professionals receive information back on the results of their 0.604 0.550 
treatment of patients 

8. Care professionals are encouraged to report incidents and adverse events 0.507 0.456 

9. Care professional licenses are reviewed by a regulatory body 0.31 6 0.304 
Formal protocols for infection control 0.79 

1. Up-to-date hospital protocol for use of prophylactic antibiotics 0.489 0.435 

2. Up-to-date hospital protocol for prevention of central line infection 0.698 0.626 

3. Up-to-date hospital protocol for prevention of surgical site infection 0.789 0.705 

4. Up-to-date hospital protocol for prevention of hospital-acquired infections 0.600 0.518 

5. Up-to-date hospital protocol for prevention of ventilator-associated pneumonia 0.659 0.576 
Formal protocols for medication and patient handling 0.77 

1 . Up-to-date hospital protocol for medication reconciliation 0.643 0.564 

2. Up-to-date hospital protocol for the handover of patient information 0.661 0.576 
to another care unit 

3. Up-to-date hospital protocol for the use of medical aids (e.g. crutches, 0.683 0.597 
bandages, etc.) 

4. Up-to-date hospital protocol for the prevention of medication errors 0.600 0.524 
Analysing performance of care processes 0.82 

1 . Root-cause analysis of incidents (an incident is an unintended event that 0.687 0.623 
has cause or could cause harm to a patient) 

2. Risk management (a systematic process of identifying, assessing and taking 0.665 0.598 
action to prevent or manage clinical events in the care process) 



(continued) 
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Table 2 Continued 



Scale and items 



Factor Internal Corrected 

loadings consistency item-total 

on primary reliability: correlation 
scale Cronbacli's 
alpha 



3. Internal audit (all components of the quality system are periodically assessed 0.543 
with regard to appropriate functioning, i.e. whether all procedures are adhered to 

and are effective) 

4. Monitoring the opinions of care professionals (physicians and nurses are 0.545 
periodically asked about their satisfaction with their work, workload, the terms 

of employment, etc.) 

5. Medical/ clinical audit (various disciplines work together to assess and 0.573 
improve the results of care delivery) 

6. Adverse event reporting and analysis (clinical staff is required to report and 0.640 
analyze all unexpected and preventable harm to patients caused by medical error 

or flaws in the healthcare system) 

7. Systematic patient record review (systematic reviews of patient records are 0.661 
used to determine adverse events and priorities for quaUt}' improvement) 

8. Development of care pathways/ process redesign (all tests and treatments for 0.520 
a specific patient group are efficientiy organized to delivery evidenced based care) 
Analysing performance of professionals 

1. Hospital (management) board 'walk rounds' to identif}r qualitjr problems and 0.549 
issues (management visits work units to discuss quality and safety issues) 

2. Monitoring individual physicians' performance (physicians undergo 0.755 
systematic and documented performance assessments) 

3. Monitoring individual nurses' performance (nurses undergo systematic and 0.650 
documented performance assessments) 

jAnaly^ngfeedhack of patient experiences 

1. Benchmarking [specific results (indicators) are compared with other hospitals 0.336 
(best in class) in order to identify possible improvement] 

2. Monitoring the options of patients (patients are periodically requested to give 0.527 
their opinions on the care provided; include surveys on patient views) 

3. Complaints analysis (periodical evaluation of complaints is used to implement 0.452 
improvements) 

Evaluating results 

\. Data used from clinical indicators to evaluate and adjust care processes 0.463 

2. Data used from complication registration to evaluate and adjust care processes 0.568 

3. Data used from incident reporting system to evaluate and adjust care processes 0.582 

4. Data used from interviews/ surveys with/ among patients to evaluate and 0.578 
adjust care processes 

5. Data used from assessment of guideline compliance to evaluate and adjust 0.601 
care processes 

6. Data used from results of internal audits to evaluate and adjust care processes 0.626 



0.72 



0.48 



0.81 



0.476 

0.497 

0.517 
0.577 

0.604 
0.460 

0.457 
0.657 
0.527 

0.219 
0.385 
0.289 



0.523 
0.629 
0.670 
0.634 

0.675 

0.726 



where a high proportion of the respondents had a score at the 
lower or upper end of the answer categories, e.g. especially for 
patient complaint analysis and monitoring patient opinions. 
More than 80-90% of the hospitals had implemented the ac- 
tivities. The overall QMSI ranged from 0 to 27 points based 
on nine scales. The mean score of participating hospitals is 
19.7 points (SD of 4.7). 



Discussion 

We set out to develop and validate an index (QMSI) to 
measure QMSs in European hospitals. We found that the 
QMSI has 46 items to be reliable and is valid for the assess- 
ment of QMSs in European hospitals. The answers to the 46 
items could be summarized in an index to express the extent 
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Table 3 Inter-scale correlation coefficients between the nine scales of QMSI {N — 181) 



Scales 


1 


2 


3 


4 


5 


6 


7 8 


1 . Quality policy documents 


1 














2. Quality monitoring by the board 


0.59 


1 












3. Training of professional 


0.40 


0.66 


1 










4. Formal protocols for infection control 


0.27 


0.20 


0.14 


1 








5. Formal protocols for medication and patient handling 


0.37 


0.41 


0.43 


0.57 


1 






6. Analyzing performance of care processes 


0.53 


0.57 


0.59 


0.24 


0.63 


1 




7. Analyzing performance of professional 


0.42 


0.52 


0.59 


0.24 


0.53 


0.65 


1 


8. Analyzing feedback of patient experiences 


0.25 


0.33 


0.25 


0.11 


0.29 


0.45 


0.27 1 


9. Evaluating results 


0.43 


0.49 


0.57 


0.14 


0.51 


0.70 


0.51 0.46 



9 



The numbers in the first row correspond with the scales in the first column. 



Table 4 Correlations of QMSI with two other measures of 
quality in a subset of 74 hospitals that studied in depth 





QMCI 


CQII 


QMSI 


0.48* 


0.34* 



*P<0.05. 



of implementation of quality management activities, such as 
quaHt}' policies, methods for continuous improvement and 
procedures for patient complaint handling or staff education. 
The QMSI was found to be useful to differentiate between 
hospitals on nine separate scales and on the index as a whole. 
The nine scales of the QMSI represent the managerial aspects 
of quaUt)' management and leave room for the investigation of 
associations of quaHtjr management with leadership, patient 
and professional involvement and organizational culture. 
These latter concepts are assumed to influence the extent of 
implementation of qualitjr management in hospitals. 



a combination of policy, monitoring quality improvement by the 
board, professional development, monitoring of performance 
of processes and knowing relevant patient-related outcomes. 

Limitations of the study 

This study has some limitations. The QMSI is based on the 
perception of the quality manager of the hospital. Although 
data from the questionnaire were self-reported, it has been 
shown through on-site visits that they seemed to be reasonably 
reliable. Despite the random selection of hospitals, selection 
bias among participating hospitals cannot be ruled out. 
Especially in some countries, the number of participating hos- 
pitals was smaller than that was initially planned for that 
country. Furthermore, the final study sample was too small to 
carry out a cross-culture validation. Therefore, further data 
collection and analysis wiU be needed before we can recom- 
mend the instrument for official use in cross-country compari- 
sons. A positive point is that hospital and country coordinators 
did not report problems with the understandabHit)' or applic- 
ability of the questionnaire. 



Comparison with earlier studies 

The newly developed DUQuE instrument has good psycho- 
metric properties, consists of up-to-date questionnaire items, 
can be used in various European countries and is not too 
time-consuming for respondents (46 items; 9 dimensions). 
Earlier developed instruments have between 17 and 179 items, 
and 3-13 dimensions [1]. The DUQuE instrument for QMS 
covers four of the six domains found in the literature. 
Intentionally, the QMSI does not cover the domains leader- 
ship and patient involvement, because these are in the 
DUQuE framework influencing factors for the implementa- 
tion of QMS and not part of the managerial aspects of quaUt}' 
management itself. 

The clear sampling frame with random hospitals across EU 
countries has a higher external validity than existing research on 
QMS. In line with previous research, it seems that there is no in- 
dividual focal area that accounts for the entire variance asso- 
ciated with the implementation of QMS. QuaUt}' management is 



Implications for research, policy and practice 

As patients and purchasers expect the best possible quality of 
care, healthcare providers have to prove that they constantiy 
work on quality improvement and safer healthcare. Our study 
has developed an efficient instrument to measure the imple- 
mentation of quality management strategies on nine focal 
areas. We also kept some items with ceiling effects, which can 
stiU support policy-makers at EU-level stimulating the devel- 
opment of QMS in less-developed countries. Areas with floor 
effects, like monitoring physician performance, are recognized 
as important for the years to come and for further spread 
in European hospitals. The instrument and resulting index 
could be used for future comparative studies on quality man- 
agement and for baseline assessment by hospitals or purcha- 
sers. A questionnaire is less time-consuming than a site visit 
and seems to give a quite reliable overall picture of the devel- 
opment and implementation of a QMS. Earlier research has 
shown that this kind of instrument is useful for monitoring 
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Table 5 Descriptive statistics of the index and nine scales of the QMSI 



Scale and items 


Possible 


Average 


Floor 


Ceiling 




range 


scores" 


C'/o with 


(% with 










lowest 


highest 










score) 


score) 


QMSI{N= 181)'' 


0- 


-27 


19.7 (4.7) 






Quality policy documents (N = 179) 


1- 


-4 


3.2 (0.8) 






1 . Written description of a formally agreed quality policy 


1- 


-4 


4(1.0) 


8 


65 


2. Quality improvement plan at hospital level (translation of the quaUtjr 


1- 


-4 


4 (2.0) 


6 


63 


objectives into concrete activities and measures designed to realize the quaUtjr 












policy) 












3. Balanced score card (an overview of key quality measures focusing on 


1- 


-4 


3 (2.0) 


18 


46 


clinical outcomes, finances, human resources and patient satisfaction) 












Quality monitoring by the board (N= 176) 


1- 


-4 


3.2 (0.7) 






1 . The hospital (management) board makes it clear what is expected from care 


1- 


-4 


3 (1.0) 


3 


39 


professionals in regards to quality improvement 












2. The hospital (management) board has established formal roles for qualitjr 


1- 


-4 


4(1.0) 


8 


66 


leadership (visible in organizational chart) 












3. The hospital (management) board assesses on an annual or bi-annual basis 


1- 


-4 


3 (2.0) 


7 


39 


whether care professionals comply with day-to-day patient safety' procedures 












4. The hospital (management) board knows and uses performance data for 


1- 


-4 


3 (1.0) 


4 


44 


quality improvement 












5. The hospital (management) board monitors the execution of quaUt}' 


1- 


-4 


3 (1.0) 


8 


41 


improvement plans 












Training of professionals (N =171) 


1- 


-4 


3.2 (0.5) 






1. Care professionals are trained by the organization to do their job 


1- 


-4 


4(1.0) 


2 


59 


2. Care professionals are trained in teamwork 


1- 


-4 


3 (2.0) 


4 


26 


3. Middle management is trained in quality improvement methods 


1- 


-4 


3 (2.0) 


4 


30 


4. Care professionals are trained in quality improvement methods 


1- 


-4 


3 (2.0) 


4 


27 


5. Care professionals are trained in patient safety procedures 


1- 


-4 


3 (1.0) 


1 


40 


6. Care professionals follow at least one training session a year to further 


1- 


-4 


4(1.0) 


2 


60 


develop their professional expertise 












7. Care professionals receive information back on the results of their 


1- 


-4 


3 (1.0) 


2 


47 


treatment of patients 












8. Care professionals are encouraged to report incidents and adverse events 


1- 


-4 


4(1.0) 


5 


53 


9. Care professional licenses are reviewed by a regulatory body 


1- 


-4 


4(1.0) 


13 


50 


Formal protocols for infection control {N =171) 


1- 


-4 


3.5 (0.6) 






1. Up-to-date hospital protocol for use of prophylactic antibiotics 


1- 


-4 


4(1.0) 


6 


71 


2. Up-to-date hospital protocol for prevention of central line infection 


1- 


-4 


4 (0.0) 


7 


73 


3. Up-to-date hospital protocol for prevention of surgical site infection 


1- 


-4 


4 (0.0) 


10 


73 


4. Up-to-date hospital protocol for prevention of hospital- acquired infections 


1- 


-4 


4 (0.0) 


3 


82 


5. Up-to-date hospital protocol for prevention of ventilator-associated 


1- 


-4 


4(1.0) 


11 


54 


pneumonia 












Formal protocols for medication and patient handling {N = 170) 


1- 


-4 


3.1 (0.8) 






1. Up-to-date hospital protocol for medication reconciliation 


1- 


-4 


3 (2.0) 


9 


45 


2. Up-to-date hospital protocol for the handover of patient information to 


1- 


-4 


4(1.0) 


5 


59 


another care unit 












3. Up-to-date hospital protocol for the use of medical aids (e.g. crutches, 


1- 


-4 


4 (2.0) 


14 


49 


bandages, etc.) 












4. Up-to-date hospital protocol for the prevention of medication errors 


1- 


-4 


3 (2.0) 


13 


48 


Analysing performance of care processes (N =168) 


1- 


-4 


2.9 (0.7) 






1. Root-cause analysis of incidents (an incident is an unintended event that has 


1- 


-4 


3 (2.0) 


10 


40 


cause or could cause harm to a patient) 













(continued) 
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Table 5 Continued 



Scale and items 



Possible 
range 



Average 
scores" 



Floor 
(% with 
lowest 
score) 



Ceiling 
(% with 
highest 
score) 



2. Risk management (a systematic process of identifying, assessing and taking 


1- 


-4 


3 


(2.0) 


12 


35 


action to prevent or manage clinical events in the care process) 














3. Internal audit (all components of the quaUt}' system are periodically 


1- 


-4 


4 


(2.0) 


8 


54 


assessed with regard to appropriate functioning, i.e. whether aU procedures are 














adhered to and are effective) 














4. Monitoring the opinions of care professionals (physicians and nurses are 


1 

1- 






\ ) 


9/1 


41 


periodically asked about their satisfaction with their work, workload, the terms 














of employment, etc.) 














5. Medical/ clinical audit (various disciplines work together to assess and 


1- 


-4 


3 


(2.0) 


15 


32 


improve the results of care delivery) 














6. Adverse event reporting and analysis (clinical staff is required to report and 


1- 


-4 


4 


(1.0) 


5 


60 


analyze all unexpected and preventable harm to patients caused by medical error 














or flaws in the healthcare system) 














7. Systematic patient record review (systematic reviews of patient records are 


1- 


-4 


3 


(2.0) 


8 


45 


used to determine adverse events and priorities for quality improvement) 














8. Development of care pathways/process redesign (all tests and treatments 


1- 


-4 


3 


(1.0) 


15 


22 


for a specific patient group are efficientiy organized to deliver evidenced based 
care) 

A^naly^^ing performance of professionals (N = 1 74) 














1- 


-4 


2.6 


(1.0) 






1. Hospital (management) board 'walk rounds' to identify quality problems 


1- 


-4 


2 


(3.0) 


27 


34 


and issues (management visits work units to discuss quality' and safety issues) 














2. Monitoring individual physicians' performance (physicians undergo 


1- 


-4 


2 


(3.0) 


37 


34 


systematic and documented performance assessments) 














4. Monitoring individual nurses' performance (nurses undergo systematic and 


1- 


-4 


3 


(3.0) 


26 


41 


documented performance assessments) 














Analj^ingfeedback of patient experiences (N= 174) 


1- 


-4 


3.4 


(0.5) 






1 . Benchmarking [specific results (indicators) are compared with other 


1- 


-4 


2 


(2.0) 


19 


31 


hospitals (best in class) in order to identify' possible improvement] 














2. Monitoring the opinions of patients (patients are periodically requested to 


1- 


-4 


4 


(0.0) 


2 


82 


give their opinions on the care provided; include surveys on patient views) 














3. Complaints analysis (periodical evaluation of complaints is used to 


1- 


-4 


4 


(0.0) 


1 


91 


implement improvements) 














Evaluating results (N = 1 77) 


1- 


-4 


3.1 


(0.6) 






1. Data used from clinical indicators to evaluate and adjust care processes 


1- 


-4 


4 


(2.0) 


3 


52 


2. Data used from complication registration to evaluate and adjust care 


1- 


-4 


3 


(2.0) 


5 


39 


processes 














3. Data used from incident reporting system to evaluate and adjust care 


1- 


-4 


4 


(1.0) 


6 


61 


processes 














4. Data used from interviews/surveys with/among patients to evaluate and 


1- 


-4 


4 


(1.0) 


1 


66 


adjust care processes 














5. Data used from assessment of guideline compliance to evaluate and adjust 


1- 


-4 


3 


(2.0) 


14 


28 


care processes 














6. Data used from results of internal audits to evaluate and adjust care 


1- 


-4 


4 


(1.0) 


3 


54 


processes 















"Median (inter quartUe range) presented for individual question items, mean (SD) presented for scales. 

''QMSI is the sum of all nine scales (minus 9). This result includes data that have been subjected to multiple imputations. 



24 



Validation of a quality management systems index 



implementation of QMSs over time [18]. More importandy, it 
can be used to test the assumption that enforcing certain quality' 
management poUcies and strategies wiU lead to the desired 
effects, possibly linking management strategies to quaHtj' and 
safet}' outcomes. Forthcoming DUQuE work wiU lend add- 
itionally validity to the QMSI through the investigation of its 
relationships with other constructs and outcomes such as hos- 
pital external assessment, quality orientation of hospital boards, 
social capital, organizational culture, safety culture, clinical indi- 
cators and patient-reported experience measures. 

Conclusion 

The newly developed and validated index (hence, instrument) 
of the implementation of QMSs presents an important tool 
for measuring, monitoring and, potentially, improving quality 
management in European hospitals. The QMSI is part of a 
broader group of instruments developed in the European re- 
search project 'Deepening our Understanding of Quality 
improvement in Europe' (DUQuE). 
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